Papers with pretraining task
The Web Can Be Your Oyster for Improving Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Pretrained language models encode a large amount of knowledge, but knowledge is frozen at the time of training, and the models become static and limited by training data. |
| Approach: | They propose an adaptive search engine assisted learning method that can self-evaluate the confidence level of PLM’s predictions and adaptively determine when to refer to the web for more data. |
| Outcome: | The proposed model outperforms retrieval-augmented methods on 16 knowledge-intensive tasks on a wide range of knowledge-related tasks. |
Curricular Next Conversation Prediction Pretraining for Transcript Segmentation (2023.findings-eacl)
Copied to clipboard
| Challenge: | Prior research on document segmentation has focused on segmenting documents such as Wikipedia articles. |
| Approach: | They propose to pretrain a model to identify consecutive conversations to address these challenges . they introduce a curriculum to Advanced NCP to make the task more relevant to the downstream task . |
| Outcome: | The proposed model outperforms previous models in speech recognition errors and is robust to speech recognition. |
A Pragmatics-Centered Evaluation Framework for Natural Language Understanding (2022.lrec-1)
Copied to clipboard
| Challenge: | a number of studies have suggested that models induce universal text representations . current benchmarks focus on semantic phenomena, so pragmatics needs to be the focus . |
| Approach: | They propose a benchmark that unites 11 pragmatics-focused evaluation datasets for English. |
| Outcome: | The proposed benchmark shows that natural language inference does not result in genuinely universal representations. |
Does Pretraining for Summarization Require Knowledge Transfer? (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing theories claim that pretraining models learn linguistic knowledge from the pretraining corpus, but scientific explanations for these benefits remain unknown. |
| Approach: | They propose to use random character n-grams to test models on real corpora to see if the small residual benefit of using real data could be accounted for by the structure of the pretraining task. |
| Outcome: | The proposed task performs on documents consisting of character n-grams, whereas pretrained models perform on real corpora with no residual benefit. |
Pseudo Zero Pronoun Resolution Improves Zero Anaphora Resolution (2021.emnlp-main)
Copied to clipboard
| Challenge: | Masked language models have contributed to drastic performance improvements with regard to zero anaphora resolution (ZAR). |
| Approach: | They propose a pretraining task that trains MLMs on anaphoric relations with explicit supervision and a finetuning method that remedies a notorious discrepancy. |
| Outcome: | The proposed method improves zero anaphora resolution in Japanese ZAR . it uses a pretrain task and finetuning task to correct the discrepancy . |
ReactXT: Understanding Molecular “Reaction-ship” via Reaction-Contextualized Molecule-Text Pretraining (2024.findings-acl)
Copied to clipboard
Zhiyuan Liu, Yaorui Shi, An Zhang, Sihang Li, Enzhi Zhang, Xiang Wang, Kenji Kawaguchi, Tat-Seng Chua
| Challenge: | Molecular-text modeling is an emerging research field that aims to facilitate molecule-relevant tasks with a textual interface and textual knowledge. |
| Approach: | They propose a new method for reaction-text modeling that uses three types of input contexts to incrementally pretrain LMs. |
| Outcome: | The proposed method improves experimental procedure prediction and molecule captioning and offers competitive results in retrosynthesis. |
End-to-end Knowledge Retrieval with Multi-modal Queries (2023.acl-long)
Copied to clipboard
| Challenge: | a new task is proposed to learn knowledge retrieval with multimodal queries . a vision-language model can retrieve knowledge using images and text inputs . |
| Approach: | They propose a task for vision-language models to retrieve knowledge with multi-modal queries . they propose reViz, a model that integrates content from both text and image queries based on a multimodal query task . |
| Outcome: | The proposed task performs better under zero-shot settings than previous work on cross-modal retrieval. |
LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for entity representations do not capture information in a knowledge base, and cannot represent entities that do not exist in the KB. |
| Approach: | They propose a pretrained contextualized representation of words and entities based on the bidirectional transformer. |
| Outcome: | The proposed model achieves impressive empirical performance on a wide range of entity-related tasks. |
SuMe: A Dataset Towards Summarizing Biomedical Mechanisms (2022.lrec-1)
Copied to clipboard
| Challenge: | Biomedical studies often examine how one entity affects another in a biological context. |
| Approach: | They propose a biomedical mechanism summarization task that pairs biomedically relevant texts with their summaries. |
| Outcome: | The proposed task improves performance but produces acceptable outputs in 32% of instances. |